]]>
<p>最近想把吃灰的<code>树莓派3B</code>拿出来用用，接通后却发现忘了密码，于是这网上找了些方法重置。</p>
<p>基本上都是这个步骤</p>
<ol>
<li>SD卡拿电脑上修改<code>cmdline.txt</code>，应该只有一行，在最后加上<code
Architecture for Distributed Python/Django/Worker Applications.http://blog.samuelchen.net/architecture-for-distributed-python-django-worker-applications/2020-06-16T15:30:52.000Z2022-02-03T19:26:34.930ZThis is the architecture I designed for service “Managed Service for Alibaba Cloud” of company production. It’s also my common architecture that can be used to deploy distributed Python applications or service based on workers or MQ. This is a easy to upgrade architecture. You can simply add other service or make it becoming microservices.
I removed some sensitive information such as source code, configuration and so on.
Requirements
OS - Linux (CentOS 7 or 6, Ubuntu ..)
Python 3.6 (< 3.7) with pip, virtualenv …
gunicorn
nginx
supervisor (if need)
DB - Postgresql (or MySQL)
Task Queue - Redis (or RabbitMQ)
source code - Removed
Install
Removed
Setup API
Removed
Architect
Note:
Nginx can be extracted on top of all boxes.
Stack Files can be stored in individual box or OSS.
For now, Celery Worker must be in API Box (for HA, should no related to API. such as create/access)
]]>
<p>This is the architecture I designed for service “Managed Service for Alibaba Cloud” of company production. It’s also my common architecture that can be used to deploy distributed Python applications or service based on workers or MQ. This is a easy to upgrade architecture. You can simply add other service or make it becoming microservices.</p>
My Contributions to Open Source Projectshttp://blog.samuelchen.net/my-contributions-to-open-source-projects/2020-06-11T13:40:03.000Z2022-02-03T19:26:34.930Z整理了自己对开源软件做的一些修改、提交或者错误报告
]]>
<p><em>整理了自己对开源软件做的一些修改、提交或者错误报告</em></p>
Window 批处理(BAT)中生成格式化日期时间字符串http://blog.samuelchen.net/generate-date-time-string-in-window-bat/2020-02-28T16:36:31.000Z2022-02-03T19:26:34.930Z今天打算写一些脚本同时支持 Windows 和 Linux，因此需要同一个脚本需要写一个 shell 版本和一个 bat 版本。其中一项需求是要生成一个日志文件，文件后缀是当前系统时间的字符串’yyymmddMMHHSS’这种格式的，所以需要取到当前时间并格式化。在 Linux 中，这是一个很简单的需求，直接使用 date +'%y%m%d%H%M%S' 就可以拿到，但是在 Windows 中就非常麻烦了。
在 Windows 中，最直接的想法是和 Linux 类似，使用 date /t 和 time /t 命令来获取，然后利用 %date:~0,4% 来取到年份（%:~0,4% 是 bat 取子字符串语法，表示取 0 到 4 个字符，前开后闭）， 其他分别取到月、日、小时、分钟和秒，最后组合。完整的例子如下.
由于 Windows 有区域和格式设定，所以 date 和 time 命令显示出来时间格式是经过格式化的，换 句话说，不同的区域由于设置的时间格式不同，返回的格式串就不同。比如中国/中文，经常使用 2020-02-02 这种格式，而欧美/英文，经常使用 02/02/20 这种格式。所以上面那个在英文系统下，多半显示不对。
也在网上找了些例子，有些考虑到了不同的格式，但也是用 date 各种判断拼凑出来，也有提到 直接访问 CMOS 硬件 clock 的方法，都是非常麻烦，不太适用于写短小的bat批处理文件。
]]>
<p>今天打算写一些脚本同时支持 Windows 和 Linux，因此需要同一个脚本需要写一个 shell 版本和一个 <code>bat</code> 版本。其中一项需求是要生成一个日志文件，文件后缀是当前系统时间的字符串’yyymmddMMHHSS’这种格式的，所以需要取到当前时间并格式化。在 Linux 中，这是一个很简单的需求，直接使用 <code>date +'%y%m%d%H%M%S'</code> 就可以拿到，但是在 Windows 中就非常麻烦了。</p>
Flask (Werkzeug) Templating Caused Trackback Loop Reference Issuehttp://blog.samuelchen.net/flask-werkzeug-templating-caused-trackback-loop-issue/2020-02-28T08:24:15.000Z2022-02-03T19:26:34.930ZFound a Flask template bug caused by PyObject_HEAD size in Python for 32bit Windows.
Today, I was facing an issue when using Flask template.
If use an dash “-“ linked variable name in Flask template, it will cause Flask to hang without throwing any exception.
At beginning, I guess it may be an issue of Jinja template rending. But after traced in IDE (Pycharm community 2019.3), I saw that templating raised the exception and Werkzeug is handling.
At first I uses Flask debug mode and IDE debug run to step into codes to dig it. IDE gets slower and slower and the debugger hung in Werkzerg codes.
Eventually, I found an endless loop in Python\Lib\traceback.py to walk trackback stacks. The trackbacks are stored in a link which nodes point to next. In flask\templating.py, flask\app.py and werkzeug\debug\tbtools.py, the exception was catched. In handle_exception() of flask\app.py, Werkzeug wants to log it and print it out. The trackback link has looped reference as below picture.
Simple code to re-produce it as below (“font-size” variable caused.).
# Mermaid (markdwon to flow chart, seq chart, class chart ...) mermaid: enable:true # Available themes: default | dark | forest | neutral theme:default # You can add more options
<!-- if CDN fails, use local file --> <script src="https://unpkg.com/mermaid@8.4.6/dist/mermaid.min.js"></script> <script> window.mermaid || document.write('<script src="/assets/js/mermaid.min.js"><\/script>')</script>
2019/05/05 14:49:44 ERROR : ... : File not in Local file system at /path/to/local/files/work/my-blog/public 2019/05/05 14:49:44 ERROR : tags/Article/index.html: File not in Local file system at /path/to/local/files/work/my-blog/public 2019/05/05 14:49:44 ERROR : tags/Config/index.html: File not in Local file system at /path/to/local/files/work/my-blog/public 2019/05/05 14:49:44 ERROR : tags/Encoding/index.html: File not in Local file system at /path/to/local/files/work/my-blog/public 2019/05/05 14:49:44 ERROR : tags/Mac/index.html: File not in Local file system at /path/to/local/files/work/my-blog/public 2019/05/05 14:49:44 ERROR : tags/Network/index.html: File not in Local file system at /path/to/local/files/work/my-blog/public 2019/05/05 14:49:44 ERROR : tags/Monitor/index.html: File not in Local file system at /path/to/local/files/work/my-blog/public 2019/05/05 14:49:44 ERROR : tags/Programming/index.html: File not in Local file system at /path/to/local/files/work/my-blog/public 2019/05/05 14:49:44 ERROR : tags/Static/index.html: File not in Local file system at /path/to/local/files/work/my-blog/public 2019/05/05 14:49:44 ERROR : tags/Template/index.html: File not in Local file system at /path/to/local/files/work/my-blog/public 2019/05/05 14:49:44 NOTICE: Local file system at /path/to/local/files/work/my-blog/public: 10 files missing 2019/05/05 14:49:44 NOTICE: Azure container $web: 10 differences found 2019/05/05 14:49:44 NOTICE: Azure container $web: 111 matching files 2019/05/05 14:49:44 Failed to check: 10 differences found # 发现 10 个不同
]]>
<pre><code>_所有静态网站都可用采用此方法_
</code></pre><p>之前，<a href="http://blog.samuelchen.net">个人博客</a>一直都是用 <code>Hexo</code> 生成并部署为 <code>GitHub Page
基于 HashiCorp Vault 和 Consul 的密钥仓库解决方案http://blog.samuelchen.net/keystore-ha-solution-by-vault-consul/2019-03-26T20:49:12.000Z2020-06-16T15:14:03.000ZKeystore cluster based on Vault & Consul
]]>
<p>Keystore cluster based on Vault & Consul</p>
<p>具体部署可参考 <a href="../encrypt-decrypt-and-store-sensitive-content-with-hashicorp-vault">敏感信息的加密存储</a> 以及 <a href="../service-discover-by-donsul">基于 Consul 的服务发现</a></p>
Encrypt/Decrypt and Store Sensitive Content With HashiCorp Vaulthttp://blog.samuelchen.net/encrypt-decrypt-and-store-sensitive-content-with-hashicorp-vault/2019-02-27T04:12:39.000Z2020-06-15T16:00:00.000ZWe are encrypting sensitive content with AES algorithm and storing them in DB. Now we plan to leverage HashiCorp Vault to do this.
So far we have 2 kinds of data need to be encrypted: one is the Alicloud key and secret of customers, the other is the marked sensitive variable or parameters such as VM password. Vault provides several secrets engines to achieve these.
Vault has a secret engine named AliCloud. But it can only manage one account with configuring a pair of its key and secret in a single path. It can only manage the policies of this account. Which means if we want store multiple customer AliCloud accounts, we need to define multiple paths in Vault for each account and configure key/secret pairs for each account manually. It’s no sense. So it’s not useful for us.
Above all, we could only use the Key/Value secrets engine.
Configuration
We define some configurations for using Vault.
ENCRYPTION_METHOD - to identify which method will be used. VAULTfor HashiCorp vault. AESfor encryption with AES algorithm and store in DB. ENCRYPTION_METHOD = “VAULT” | “AES”
VAULT_KV_PATH - Vault path is related to secrets engine. This need to defined from architecture level so that all components which uses Vault will follow it. And avoid namespace conflict. By default, the base path for Key/Value engine is “kv”. e.g. VAULT_KV_PATH = “kv/cpsapi”
NOTABLE IMPLEMENTATION details
To distinguish encryption methods for the encrypted data, we add prefix for it. e.g. for AES, it will be “**$:AES:$**qowzobv1GoC8tB7dOrHJpA==PXyKO2kLX6C4oD8a+e5BlMQwD2q8vvu3poI5rishmVM=” . If data encrypted by Vault, the encrypted data will not be stored in DB. So that it will only remains “$:VAULT:$”
Two utility methods wrapper encrypt/decrypt were created. In them, we automatically check which methods will be used and apply it. If the ENCRYPTION_METHODS was changed after the system runs for some while, we may be facing some data was encrypted by AES but now the decryption method is VAULT. For this scenario, we have 2 choices:
Decrypt the data as it was. Encrypt it as configured.
Raise an exception to notify. This means if a encryption methods is used when the system initializing, it can not be changed.
Data length changed. Encrypted data will has a different length compares to origin data. So the DB field length need to be changed. If the ENCRYPTION_METHODS changed, we have 2 choices:
Change the field max length to max of all encrypted data length. e.g. Origin field length is 10. AES encrypted data length is 64. VAULT encrypted data length is 9. The we choose max(64, 9) = 64.
Use the encrypted data length as configured methods and Raise an exception to notify. “ENCRYPTION_METHODS can not be changed.”
Another problem for data length changing is, in admin panel, the data length validation is not valid anymore. So far no good way to resolve (No place to store the origin length in DB) If we only allow VAULT, this and above 2 issue may not existed. (we keep the filed with fixed length. we do not add prefix.)
Each access to sensitive data will cause an REST API call to Vault server. This could be potential performance issue.
— END —
]]>
<p>We are encrypting sensitive content with AES algorithm and storing them in DB. Now we plan to leverage HashiCorp Vault to do this.</p>
Solution Selection for Sensitive Content Storinghttp://blog.samuelchen.net/solution-selection-for-sensitive-content-storing/2019-01-11T11:10:38.000Z2020-06-15T16:00:00.000ZIn “Managed Service for Alibaba Cloud” service project, we have some sensitive data such as secret key, vm password and so on stored in database. To avoid store plain text, we need a solution to encrypt and decrypt sensitive data.
Solutions
Keystore
Systems that provide ability to store sensitive key/token/password. Encryption and decryption will handled by the system.
IDM
Valut
Need to involve a new system and maintain it.
Encoding
Reversible character encoding algorithms. Same algorithm will be used for encryption and decryption.
HEX, BASE64 (BASE##), UTF8 and etc.
compaction algorithm such as Huffman Coding, ZIP (Deflate) , JPEG, MPEG and etc.
Reversible algorithm. Plaint text (data) is encrypted with public key. The encrypted data is decrypted only with private key. The public key can be openly distributed without compromising security. Encryption and decryption in different way. https://en.wikipedia.org/wiki/Public-key_cryptography
RSA
DSA
ECC
etc.
Anaysis
Keystore - One more system means one more point need to care the load, failure and so on. Need to communicate between severs even to set/get a property of resource (such as vm property password).
Abstract algorithm - Irreversible. Not appropriate.
Symmetric encryption - Good choice. Need to store the key in separate place. Good solution for encryption and decryption in same system. (e.g. AES)
Asymmetric encryption - Good choice. Need to maintain 2 keys and store in separate places. It’s not necessary because we have the encryption & decryption in same application. Good solution for separate encryption and decryption. (e.g. RSA) For RSA, plain text can not be longer than the key.
Maturity
Security
Speed
Load
Comment
DES
High
Low
Medium High
Medium
AES
High
High
High
Low
Replacement of DES
RSA
High
High
Low
High
plaint text can not longger than key
DSA
High
High
Low
Only for digital signature.
ECC
Low
High
High
Low
Above all, we better choose Symmetric encryption AES. And AES is faster and lower load compares to RSA.
Resolution
The database server and application server are behind the firewall. And they are in private network. This ensures the architecture level security.
Key
Stores in OSS or shared NFS with authorization. Read when application started (only in memory).
Customer Alicloud key & secret
Encrypt it before saving. Decrypt it after loading.
Virtual Instance password in parameter
Encrypt it before saving. Decrypt it after loading.
— END —
]]>
<p>In “Managed Service for Alibaba Cloud” service project, we have some sensitive data such as secret key, vm password and so on stored in database. To avoid store plain text, we need a solution to encrypt and decrypt sensitive data. </p>
为2012年中的13寸macbook pro更换固态硬盘SSD并添加内存http://blog.samuelchen.net/Change-SSD-and-add-memory-for-2012-mid-macbook-pro/2018-11-14T16:55:37.000Z2022-02-03T19:26:34.930Z刚刚给2012年中的13寸MacBook Pro更换了SSD固态硬盘，并且添加了内存，记录一下。
/dev/disk2s1 on / (apfs, local, journaled, noatime)
— END —
]]>
<p>刚刚给2012年中的13寸MacBook Pro更换了SSD固态硬盘，并且添加了内存，记录一下。</p>
API Gateway by Fabio & Consulhttp://blog.samuelchen.net/api-gateway-by-fabio-n-consul/2018-02-28T08:21:23.000Z2020-06-15T13:42:12.000ZFabio is a HTTP router app written by Go language. It is Zero-configuration and simple to deploy. As official document mentioned, “It delivers 23.000 req/sec every day since Sep 2015 without problems”. It’s written and maintained by Frank Schroeder at eBay in Amsterdam.
Fabio is a single file app. You just download the executable for your OS from its release page (https://github.com/fabiolb/fabio/releases) or use brew install fabio (MacOS) or go get github.com/fabiolb/fabio (Golang) to download to local. Rename it to fabio and add executing privilege.
Configuration
Fabio is zero-conf app. Which means you need almost no configuration. If you deploy it on the machine where you Consul deployed, it will automatically find service and register health check on it (Consul default localhost:8500)
Note that “tags”: [“propel-/jekins”]. It means this is a service for Fabio. propel- is the prefix of tag to identify (which default is urlprefix-. You can change it by -registry.consul.tagprefix “propel-”.) /jekins means the endpoint to the service.
Visit http://myconsul:8500/ and check “service” tab, you will see registered service and the tags.
Visit http://myfabio:9998/, you will see the routers (with red rectangle marked)
You need to set the tag with an option strip=/tomcat. It means the router will strip /tomcat from url path when routing. See the blue rectangle mark in the above 2nd screenshot (option column of tomcat). You will see it in Consul dashboard as below.
You may use –help to check the arguments. Or use a fabio.properties file as configuration.
To run with prefix, $ fabio -registry.consul.tagprefix propel-
Run as service
TODO
Other
When I test with Apache Benchmark (ab), if the concurrent clients number sets to 500, it will occurs socket too many open files error. You need to use ulimited -n 65535 to fix it.
Check the log, you will see which service is captured.
— END —
]]>
<p>Fabio is a HTTP router app written by Go language. It is Zero-configuration and simple to deploy. As official document mentioned, “It delivers 23.000 req/sec every day since Sep 2015 without problems”. It’s written and maintained by Frank Schroeder at eBay in Amsterdam.</p>
Machine Learning Memo 1http://blog.samuelchen.net/Machine-Learning-Memo-1/2017-12-12T07:19:20.000Z2022-02-03T19:26:34.930Z最近开始看Andrew Ng 的 Machine Learning 系列课程，记录一下体会。
Machine Learning 学习笔记 - 第一周
重要关键词：Supervised, Unsupervised, Regression, Classfication, Features
]]>
<p>最近开始看Andrew Ng 的 <a href="https://www.coursera.org/learn/machine-learning" target="_blank" rel="noopener">Machine Learning 系列课程</a>，记录一下体会。</p>
How to Setup Kafka Web Consolehttp://blog.samuelchen.net/how-to-setup-kafka-web-console/2014-04-24T07:15:11.000Z2022-02-03T19:26:34.930Z“Kafka Web Console” is a open source monitor tool for Kafka.
“Database xxx needs evolution!” - Solution for database is not initialized.
use “play -DapplyEvolutions.default=true” to start.
Or add “applyEvolutions.default=true” in conf/application.conf
— END —
]]>
<p>“Kafka Web Console” is a open source monitor tool for <code>Kafka</code>. </p>
Ubuntu12 虚拟机网络配置http://blog.samuelchen.net/ubuntu12-vm-network-config/2012-06-21T11:09:05.000Z2022-02-03T19:26:34.930Z后来，了解了更多的方法，这种方法并不是一个比较好的方法，但仍不失为一个简单有效的方法。
用VM Player 4.0装了Ubuntu 12 server，然后复制了几个组成局域网来使用，发现了一些问题，网上基本上找不到完整的解决方法，这里记录下来。
云端现在是Amazon, Google暂时领先于基础架构领域，MS, Apple 处于追赶之中，Facebook/Twitter则领先于SNS（此SNS含义相当广，包括身份，信用，关系，服务，应用等等）方面。传统厂商如IBM, HP基于其固有的优势，大力推广所谓私有云，实质是企业cluster，在下一代互联网中已经落后一步。
而终端则是Apple，Google 和 MS 三足鼎立。iOS和Android，已经成为了移动领域事实上的领头羊，MS在桌面的固有优势以及家庭媒体中的发力，使其保持在第一梯队中。而HP，虽然购买了3Par, Palm/webos，但依然未能表现出足够的进取。其他厂商，目前看来，只是陪太子读书的角色。
